Collaboration system

ABSTRACT

A collaboration system provides enhanced user interface to enable users to interact with electronic devices. In one embodiment, users can add content to a digital system by using a pen that streams coordinates so that input to the digital system may be based on conventional pen and paper handwriting. In another embodiment, a pie-based menu system is used for input to large display area digital devices in which an occluded portion of the pie-based menu system is not used for direct input by the user. The selection of which areas of the pie-based menu system should be omitted from use is adaptive and responsive to whether the user is left-handed or right-handed, and the wrist angle defined by the user&#39;s posture. In still another embodiment, an ergonomic open-shaped pie menu system is provided to facilitate selection of options on a digital surface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.12/567,481 entitled “Collaboration System” to Doray et al. filed Sep.25, 2009, which claims priority to U.S. Provisional Patent ApplicationNo. 61/100,066, filed Sep. 25, 2008, entitled “Occlusion—Aware MenuDesign for Digital Tabletops”, both of which are hereby incorporatedherein by reference.

TECHNICAL FIELD

The present invention relates to collaboration systems and, moreparticularly, to a method and apparatus for facilitating collaboration.

BACKGROUND

Data communication networks may include various computers, servers,nodes, routers, switches, bridges, hubs, proxies, and other networkdevices coupled together and configured to pass data to one another.These devices will be referred to herein as “network elements.” Data iscommunicated through the data communication network by passing protocoldata units, such as data frames, packets, cells, or segments, betweenthe network elements by utilizing one or more communication links. Aparticular protocol data unit may be handled by multiple networkelements and cross multiple communication links as it travels betweenits source and its destination over the network.

Data communication networks are frequently used to interconnect peopleto enable them to collaborate on projects from different geographiclocations. For example, in a business context, people from a givencompany may be physically located at different offices but may beworking together on a common project. Thus, the people will need tocollaborate with each other to discuss problems associated with theproject and to collectively find solutions to the problems.

Data communication networks facilitate collaboration in many ways.Initially, communication networks were used to convey voice betweenparticipants so that multiple people could talk to each other on aconference call. Over time, the basic voice conference call has beenextended to also include video conferencing and other types ofconferencing systems.

When people meet to collaborate on a project, it is possible for thepeople to become distracted and unfocused. Accordingly, various businesstools have been developed to help lead business meetings to helpparticipants stay focused on the intended topic. Examples of businesstools of this nature include whiteboards and flip charts. These businesstools, when properly used, can help focus a group of people on a smallset of topics by providing a way for the moderator of the meeting tokeep returning the participants' attention to the main theme of themeeting. For example, by interrupting and requesting a person tosuccinctly state their thought, and then writing the thought on thewhiteboard, the moderator may focus the person's attention and return tothe main objectives of the meeting. Whiteboards may also be used toenable people to brainstorm solutions to a particular problem, vote onthe various solutions, and otherwise record the content of the meeting.Myriad uses for white boards have and may be developed.

When not everyone is in the same room, it is possible for the remoteparticipants to look at an electronic whiteboards which may be run as anapplication on a personal computer. The electronic whiteboard may beused in a manner similar to conventional physical whiteboards, exceptthat the participants are not required to be in the same room.

Whiteboards have also changed from being flat white boards on whichparticipants could write using erasable markers, to the point where theyare now being implemented using large digital surfaces. These same typesof large digital surfaces are also being used to implement tabletopsurfaces or other horizontal surfaces to let users input and viewcontent on the table rather than requiring the users to look up at awall. As these systems become more prevalent, it would be advantageousto provide a manner for remote participants to collaborate with peoplegeographically located with the large digital surface. Similarly, itwould be advantageous to provide a more optimal way for users to accessfunctions associated with programs running on the large digital surface.

SUMMARY

The following Summary and the Abstract set forth at the end of thisapplication are provided herein to introduce some concepts discussed inthe Detailed Description below. The Summary and Abstract sections arenot comprehensive and are not intended to delineate the scope ofprotectable subject matter which is set forth by the claims presentedbelow.

A collaboration system provides enhanced user interface to enable usersto interact with electronic devices. In one embodiment, users can addcontent to a digital system by using a pen that streams coordinates sothat input to the digital system may be based on conventional pen andpaper handwriting. In another embodiment, a pie-based menu system isused for input to large display area digital devices in which anoccluded portion of the pie-based menu system is not used for directinput by the user. The selection of which areas of the pie-based menusystem should be omitted from use is adaptive and responsive to whetherthe user is left-handed or right-handed, and the wrist angle defined bythe user's posture. In still another embodiment, an ergonomicopen-shaped pie menu system is provided to facilitate selection ofoptions on a digital surface.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the present invention are pointed out with particularity inthe appended claims. The present invention is illustrated by way ofexample in the following drawings in which like references indicatesimilar elements. The following drawings disclose various embodiments ofthe present invention for purposes of illustration only and are notintended to limit the scope of the invention. For purposes of clarity,not every component may be labeled in every figure. In the figures:

FIG. 1 is an example network that may be used to implement acollaboration system according to an embodiment of the invention;

FIG. 2 is a flow diagram illustrating the flow of information in thereference network of FIG. 1 according to an embodiment of the invention;

FIG. 3 is a two dimensional representation of a menu design for use witha large digital surface designed for a right-handed user according to anembodiment of the invention;

FIG. 4 is a two dimensional representation of the menu design of FIG. 3in greater detail according to an embodiment of the invention;

FIG. 5 is a two dimensional representation of a menu design for use witha large digital surface designed for a left-handed user according to anembodiment of the invention;

FIGS. 6A-6D show adaptations of a menu according to the user's postureaccording to an embodiment of the invention;

FIG. 7 is a two dimensional representation of the menu showingrelocation of the menu design depending on location of an input deviceaccording to an embodiment of the invention;

FIG. 8 is a two dimensional representation of a large digital surfacewith multiple left and right-handed menu designs implemented formultiple users according to an embodiment of the invention;

FIG. 9 is a two dimensional representation of another menu design foruse with a large digital surface designed for a right-handed useraccording to an embodiment of the invention;

FIG. 10 is a two dimensional representation of yet another menu designfor use with a large digital surface designed for a right-handed userand incorporating a scroll function according to an embodiment of theinvention;

FIG. 11 is a two dimensional representation of a menu design for usewith a large digital surface illustrating display of sub-menus;

FIG. 12 is a two dimensional representation of a menu design for usewith a large digital surface illustrating display of one sub-menu overlayer on top of a main menu;

FIG. 13 is a graph showing the range of movement of a user's hand whenresting on a surface such as a large digital surface;

FIG. 14 is a graph showing the range of angle movement of a user's handwhen resting on a surface such as a large digital surface

FIG. 15 is a graph showing acceptable area for display of selectableicons in a menu design for use with a large digital surface according toan embodiment of the invention;

FIG. 16 shows a selection of selectable pre-defined differently sizedacceptable areas within which menus may be drawn for users in anergonomic manner according to an embodiment of the invention;

FIG. 17 shows a process that may enable a user to provide input to thelarge digital surface to specify a size of an acceptable input areawithin which a menu may be drawn for the user in an ergonomic manneraccording to an embodiment of the invention;

FIG. 18 shows an open pie menu according to an embodiment of theinvention that has been drawn within a reachable area for a user.

FIG. 19 shows an embodiment of a menu design with two rows ofpoint-and-click regions.

FIG. 20 shows an embodiment of the invention in which a modified piemenu of FIG. 4 has been drawn within the reachable region surrounding aninitial pen placement;

FIG. 21 is a functional block diagram of a collaboration server designedto implement at least a portion of the collaboration system according toan embodiment of the invention; and

FIG. 22 is a functional block diagram of a large digital surfacedesigned to implement at least a portion of the collaboration systemaccording to an embodiment of the invention.

DETAILED DESCRIPTION

FIG. 1 illustrates an embodiment of the invention in which an electronicpen is used to enable users to interact with other users in acollaborative manner. Specifically, as shown in FIG. a person using anelectronic pen 50 is remotely located from other participants and,hence, cannot directly interact with those participants. According to anembodiment of the invention, the electronic pen 50 can enable the userto interact with the other participants 52 in a collaboration center 55over a network 54.

As shown in FIG. 2, the pen user uses the electronic pen 50 to write ona special pad of paper or on ordinary paper depending on the type ofelectronic pen being used. The written content is stored in theelectronic pen and, on demand, streamed from the pen to a collaborationserver 56. Where the pen is configured to stream the information overthe network to the collaboration server directly, it may do so.Alternatively, the electronic pen may stream the information to acomputer 58 or other handheld electronic device associated with the penuser which may process the data and/or relay the data to thecollaboration server 56. The collaboration server 56 interprets the penstrokes from the electronic pen and provides an updated output to theother participants 52 participating in the collaboration session bycausing the updated content to appear on a large digital surface 57physically present with the other participants in the collaborationcenter 53. For example, if the electronic pen user draws an image, theimage may be shown to the other participants on the large digitalsurface in the collaboration center. Alternatively, if the pen userinvoked a function such as voted on an item, the pen user's input willbe provided to the other participants to show an updated collaborationview.

In one embodiment, an electronic pen such as an Anoto™ Pen may be usedto interact with the collaboration system. Anoto pens use special padsof paper and optical recognition to determine where the pen is on apiece of paper and stream coordinates (wirelessly or via a USB cable)using a proprietary format. Software on the computer receives thecoordinates and translates the pen motion into text or graphics.Alternatively an IOGear pen such as the Mobile Digital Scribe™ may beused to electronically stream coordinates of the pen strokes (which aremade on paper) to a computer. The Mobile Digital Scribe will storehandwritten notes, drawings, etc., for later transfer to a computer forstorage or rendering.

The electronic pen has an ink reservoir and a nib that enables ink to bedispensed while the user moves the nib across the paper. Thus, the useris provided with feedback in the form of lines on the paper. In thisregard, the pen operates as a normal pen such that the user may use thepen to draw figures and words on normal paper. However, the user's penstrokes are also captured electronically and transmitted on a network tocollaboration software that enables the user's input to be captured andprovided to the other participants. Likewise, the user's input may berecorded if the session is being stored, so that the user's input may bememorialized without requiring further action by the user or otherparticipants.

For example, an electronic whiteboard application may be used as ashared application which may be viewed and interacted with byparticipants. The participants can be local, i.e. located in the roomwith a large digital surface showing an instance of the electronicwhiteboard, or may be remote and interface with the electronicwhiteboard over a network. In one embodiment participants m acollaborative session can interact and share content electronicallythrough the use of an electronic pen and paper. Participants write onpaper with an electronic pen. The pen streams the coordinates of the penstroke corresponding to what is being written on paper to acollaboration server. The computer system can then share this writteninformation to the participants connected to the same collaborativesession via a computer or some other electronic device. For example, thecoordinates may be received and transferred on a wireless network by theuser's handheld wireless device (e.g. cell phone, Blackberry, PersonalData Assistant PDA).

When the written content is received, it can be shared with the otherparticipants either in real-time or only when the pen-user taps on aspecially designated region of the paper sheet or form.

Use of an electronic pen as an input mechanism is simple and natural forusers, since writing using pen and paper is ubiquitous and familiar formost individuals. Additionally, when compared with the use of a tabletPersonal Computer (PC), the use of an electronic pen is much more costeffective since table PCs are expensive to purchase and maintain. Bycontrast, the electronic pen may connect to an existing computer orhandheld wireless device, so that the upfront and operational costsassociated with tablet PCs may be eliminated.

In one embodiment, the user's input is not shared in the collaborationsession with the other collaboration participants until the userinitiates sharing. For example, the electronic pen user may need to tapa particular area of the paper before the content is streamed to theother users. This allows the user to complete a drawing or thoughtbefore streaming any content to the other people joined in thecollaboration session, to avoid interrupting the collaboration sessionwith partially completed content. When the user initiates sharing, thepen strokes are streamed from the user's pen to a computer system andstored there. The computer system then provides the pen-strokes eitherdirectly or in synthesized form to the other participants that arecollaborating with the electronic pen user to enable the otherparticipants to see what the electronic pen user has drawn.

There are many uses of this electronic system. For example, in a votingscenario, a ballot form will generally have well defined regions thatcorrespond to each choice for the vote. When the user taps on one of thechoices, the electronic pen can stream the pen coordinates to thecollaboration system which translates the coordinates into a votingchoice. The voting choice will be registered by the collaborative systemand tallied with the votes from the other participants in thecollaborative session.

As another example, in traditional brainstorming sessions participantsare given sticky “post-it” notes to write down their ideas. These ideasare then stuck on a wall and physically moved around to form groups ofrelated ideas. At the end of the brainstorming session these ideas arethen manually typed into a computer for sharing. This way of workingmakes it hard to include remote participants since the remoteparticipant must verbally relay his idea to a local participant so thatthe local participant can create the paper post-it note that is thenadded to the wall. Using an electronic pen, the user can write theirideas on specially marked note pads with an electronic pen. When theuser taps on specially marked areas on the note page, the idea isuploaded to a virtual wall implemented as a large digital surface. Theidea may even be represented on the virtual wall using a virtual“post-it” note. Once the idea is shown on the virtual wall (along withthe other ideas that have been posted on the virtual wall) theindividual ideas can be moved around with a mouse, light pen, touch,etc., to group similar ideas in a manner similar to what could beaccomplished using physical notes in a normal brainstorming session.

Large direct-input tables (horizontal surfaces) and large direct inputwalls (vertical surfaces) have been developed to enable multiple usersto simultaneously provide input to facilitate collaboration. Bothhorizontal and wall mounted surfaces will be described herein as “largedigital surfaces”. Large digital surfaces can be optical and/or pressuresensitive to allow users to interact using light pens and/or by touchingthe surfaces.

Computers are good at facilitating collaboration between people inremote locations, but when people are grouped together computers tend toget in the way of collaboration. Large digital surfaces such as theDiamondTouch table available from Circle Twelve™ facilitatecollaboration by enabling multiple users to interact with the samecontent rather than having each user looking at the content on their ownscreen. The DiamondTouch table, for example, is a large direct-inputtable that can distinguish between touches from multiple userssimultaneously, thus allowing multiple users to add content to thatbeing shown by the large digital surface.

Frequently, one or more programs will be run on a large digital surface,and users invoke functions on these programs by interacting withapplication menus. The large digital surface itself may also have one ormore available functions as well, and thus may have its own menu system.Thus, users of large digital surfaces will often need to access menus onthe large digital surface.

Unfortunately, on large digital surfaces, occlusions created by theuser's hand decrease interaction performance with menus. Menus arecommonly used to access features available in programs running on acomputer system associated with the direct input surface. According toan embodiment of the invention, menus are drawn to avoid the occlusionareas caused by the interaction between the user's hand and the largedigital surface. The menus are adaptive according to user's handedness,orientation, and position on the tabletop. The menus may include pointand click areas as well as gesture input areas. Optionally, sinceinteraction with the gesture area does not rely on the user's ability tosee the menu, the gesture area may be partially or wholly containedwithin the occlusion area.

Interaction with large direct digital surfaces is strongly influenced byphysical restrictions. Reachability of items and obscurations caused bythe user's body relative to the surface may make particular menu designsdifficult to use. For example, traditional menus are not very welladapted to direct pen or direct hand-screen interaction. Menus thatappear on the location where they are activated seem to be a betterchoice for large interactive surfaces, where the input is normally donewith a pen or a direct finger touch. Additionally, different users mayprefer to use their right hand to interact with menus, while other usersmay prefer to use their left hand to interact with menus. This innatepreference will be referred to herein as “handedness”. The handedness ofthe user will affect which areas of the large digital surface areobscured when the user attempts to interact with a menu. Likewise theuser's posture, which affects the direction in which the user's hand isoriented on the large digital surface while interacting with the menu,will affect which areas of the large digital surface are occluded.

FIG. 3 shows an example menu adapted for use on a large digital surfaceaccording to an embodiment of the invention. As shown in FIG. 3, themenu is configured to incorporate some of the features of a conventionalpie menu. However, conventional pie menus extend 360 about a centralarea. According to an embodiment of the invention, a region of the piemenu that is likely to be occluded by the user's hand is not used forpoint-and-click interaction.

For example, as shown in FIG. 3, the menu 10 includes a point and clickregion 12 and a gesture area 14. The point and click region 12 may bedivided logically into a plurality of areas as shown in FIG. 4. Each ofthe areas may include an icon 18 or other indicia of the functionavailable via that area may be presented to help instruct the user ofthe function of that point and click area. In operation the user may usea pen 20 or their finger to select one of the icons to cause thefunction to be executed or to cause a sub-menu to be drawn to make afurther selection of functions available to the user.

In the embodiment shown in FIG. an occluded area 22 which is expected tolie beneath the user's hand while the user interacts with the menu 10 isnot used to support point and click interactions between the user andthe underlying large digital surface. The occluded area, or a portion ofthe occluded area, may however be used to support an extended gesturearea. The gesture area may be used by users to draw lines, select icons,cause new icons to appear, and in other ways. Gesture based iconselection and gesture based scrolling is known in the art and theparticular manner in which the gesture area is used would depend onsoftware that the menu was being used to interact with and the type offeatures/functions available via the menu. Specifically, the menudescribed herein and illustrated for example in FIG. 4 may be used witha variety of different software programs, and the particular functionssupported by the icons would change depending on the functions availablein the program. Use of the gesture area would similarly be expected tobe correlated with the application so that different gestures may beused differently to interact with different applications in use via thelarge digital surface.

The occluded area 14 may be used as part of an interactive area forgesture input inside the menu. For example, as shown in FIG. 3 thegesture area may include an extended gesture area 14′ which extends intothe occluded area 22. Occlusions prevent visual feedback but this is nota problem for gesture based input as long as the gesture area can berecognized, the user knows where he can start a gesture, and the userknows which gestures to use. The outer region of the menu should be usedfor the items which can be accessed with a simple point-and-click Oneexample for a gesture-based metaphor is a technique in which thecurvature of the path created by the pen is tracked and used to scrollcontent or zoom into the scene, for example. This method tracks thecurvature of the path created by the pen and changes the valuesaccordingly. The speed at which the users browse through content iscontrolled by the radius of the circle drawn: the larger the radius, theslower the increase in the controlled parameter, and the slower thespeed of the browsing of content. In the same way, users can changeother parameters like stroke width. Another example of a gesture may beto enable users to change pages by sliding the pen from the left to theright in the gesture area of the menu.

According to an embodiment of the invention, the orientation of the menuadjusts to make the menu adaptive, so that the occluded area is orientedin the direction of the user's hand when the user is interacting withthe large digital surface. FIGS. 4 and 5 show a first way in which theuser's hand orientation is used to adjust the location of the occludedarea 22. Specifically, FIG. 4 shows a menu that may be drawn for aright-handed user and FIG. 5 shows the same menu when drawn for aleft-handed user. As is clear from a comparison of these two figures, aright-handed user is a user that interacts with the large digitalsurface using their right hand. Accordingly, the area of the largedigital surface that will be occluded by a right-handed user will be thearea in the lower right-hand area of the circle. Likewise, the area ofthe large digital surface that will be occluded by a left-handed userwill be the area in the lower left-hand area of the circle. FIGS. 4 and5 show how the menu is adjusted, according to an embodiment of theinvention, to accommodate the handedness of the user.

Large digital surfaces such as the DiamondTouch surface are touchsensitive. When interacting with a menu displayed on a horizontal flatsurface such as a desk, many people will rest their hand or elbow on theflat surface. According to an embodiment of the invention, theorientation of the menu may be further adjusted to accommodate differentpostures by comparing the location where the user places their wrist orelbow with a location where the user touches the pen to the largedigital surface. Where the user does not rest their hand, the menuorientation may be based on a default direction vector value or mayresort to a previous orientation based on a direction vector determinedduring a previous menu invocation operation when the user did rest hishand on the large digital surface.

FIGS. 6A-6D show an example of how this may occur. As shown in FIG. 6A,the user's wrist contacts the large digital surface at point W and thepen contacts the large digital surface at point P. A vector V betweenthese two points may be used to adjust the orientation of the occlusionarea 22 such that the occlusion area is relatively constantly locatedrelative to the vector. Note, in this embodiment, that the menu is drawnto be centered on the place where the pen contacts the digital surface.By causing the occlusion area to be relatively constantly oriented aboutthe vector from the wrist location to the pen location, the user's handshould be approximately located over the occlusion area 22 regardless ofhow the user holds the pen and regardless of the user's wrist posture.

The menu is centered on the pen's position and, in this embodiment,rotated with the information of the direction vector. This provides twoadvantages: First, this allows the menu to automatically adapt for leftand right-handed users, as the menu rotates according to the directionvector from hand to pen. Second, the orientation is correct from anyperspective on the tabletop and occlusions are avoided. If the user doesnot contact the tabletop with their wrist, another method such asshadow-tracking could also be used to determine a direction vector.Alternatively, a default direction vector or previous direction vectormay be used to establish the orientation of the menu for the user.

FIG. 7 shows an example of how the large digital surface can present theuser with a menu. Specifically, as shown in FIG. 7, the menu 10 will bedrawn at a location surrounding the user's pen tip when invoked. Thus,for example, if the user invokes the menu at a first location, movestheir pen, and invokes the menu a second time, the menu will be drawn tosurround the pen-tip where ever invoked by the user.

A pen may be used by a user for many purposes, such as to draw pictureson the large digital surface, write text, add notes, move items, etc.Most of these operations do not require the menu to be invoked.Accordingly, the simple act of touching the pen to the table surfaceshould not be interpreted as an instruction to invoke the menu. However,there are times when the user will need to invoke the menu.

If the pen has an integrated button, this can be used to achieve thesame effect as a right mouse button click. But as not all pens offer anadditional button and, hence, it should be possible to invoke a menuwithout relying on using a special button on the pen. The double clickinvocation is well known from mouse based applications. The importantcharacteristics for a double click are two down events in a certainlapse of time and within a spatial area. Unfortunately, basing a menuinvoke operation on two down events when using a pen can ambiguous,depending on the application context in which the double click is usedto active a pie menu. For example, if handwriting is the main task withthe pen, it can be difficult to distinguish between a double-clickmenu-invoking event and writing colons on the page.

According to an embodiment of the invention, the double-click (two downactions with a pen) is extended so that, with a pressure-sensitive peninteraction, rather than using two down motions to invoke a menu, asingle down event followed by two presses without an interveningup-event is used to invoke the menu. In this embodiment each press onthe pen tip has to exceed a predefined threshold to be recognized. Aswith the double click, the sequence has to be performed in a certainlapse of time and within a spatial area. One advantage of doublepressure over double click is the more stable position of the pen duringthis action. Double click requires lifting the pen between the twoclicks, double pressure assumes the pen is at a fixed position withinsome tolerance. Although the double pressure event described herein hasbeen described in connection with invoking a menu, it may be possible touse this type of interaction more generically to invoke other actionsother than activating the pie menu.

FIG. 8 shows an example large digital surface embodied as an electronictable. Four users are spaced around the large digital surface, each ofwhich has its own pen. Some digital surfaces have pens assigned toparticular locations while others enable pens to be assigned toparticular users. When each user sits down at the electronic desktop,the user may initiate use of the pen to specify, for example, whetherthey are left-handed or right-handed. Each user may then invoke a menuby utilizing an invocation action as described above. For example, FIG.8 shows 4 users simultaneously invoking menus in their respective areaof the electronic desktop. In this example, users 1, 3 and 4 areright-handed and user 2 is left-handed.

Digital pens such as Anoto pens have an ID associated with them that maybe recognized by the large digital surface. Upon invocation, the largedigital surface will draw a separate menu for each pen individually.Hence, multiple users may interact with the large digital surfacesimultaneously. Due to the adaptive placement (use of vectors to orientthe menus), the menu will be oriented towards the user independent ofthe user's position around the large digital surface. Each pie menustores its user's specific settings and, when the user invokes the menu,the menu will be drawn using these stored values.

FIG. 9 shows another example menu 10 having the same basic structure asthe menu of FIGS. 4 and 5, and including a point-and-click area, agesture area, and an occluded area 22. As with the embodiment shown inFIGS. 4-5, the occluded area causes the point-and-click area to beimplemented as a circle segment rather than as a full circle so thatpoint-and-click regions are not provided in an area that is expected tobe below the user's hand as the user interacts with the menu.

The example shown in FIG. 9 further includes an arrow 24 within theoccluded area that may be used to cause the icons shown in the regions16 of the point-and-click area 12 to scroll. Scrolling as used hereinrefers to causing the icons to rotate around the circle to cause theicons to occupy different regions within the point and click area of themenu. FIG. 10 shows another embodiment where the scrolling action iscontrolled using a spiral gesture in the gesture area 14.

FIGS. 9-10 show conceptually how icons can scroll through the regionsaround the segment forming the point and click area 12. Specifically, inFIG. 9, icon #1 is displayed in region 16A. After interaction such as bytaping the pen inside the region defined by the arrow 24 or drawing acircular gesture in the gesture area 14, the functions have scrolledsuch that icon #1 has been from its location in region 16A in FIG. 9 toregion 16B in FIG. 10. All of the other icons have similarly been movedcounterclockwise by 18 positions. In this example, as shown in FIG. 10,there are eighteen icons that may be displayed in the regions 16 of themenu 10. Other numbers of icons could be used. By scrolling thefunctionality, a larger number of functions may be made available via asmaller number of regions 16.

FIGS. 11 and 12 show example displays of sub-functions that may beinvoked when a user selects an icon in one of the point-and-clickregions. The difference between FIGS. 11 and 12 is in how thesub-function menu 26 is drawn. Specifically, in FIG. 11 the sub-functionmenu 26 does not overlap with the main menu whereas in FIG. 12 thesub-function menu is overlayed on top of the main menu. The sub-functionmenu may extend up, down, to the side, or may be centered over the mainfunction region 16 used to invoke the sub-function menu. Optionally theregion associated with the main functional icon may be highlighted tofurther visually show the user which icon has been selected and, hence,enable the user to visually correlate the main function with thesub-functions available via the sub-function menu.

FIGS. 4-5 and 9-10 show pie-shaped menu embodiments implemented to havethe direct pen input area (point and click area 12) extend differingamounts around the circle. Direct pen input areas in this explanationare areas where a user can use the pen to select a particular function.Indirect pen input areas, such as gesture areas, are in contrast withdirect pen input areas and are areas where the user cannot invoke aparticular function but rather the user can control which functions areavailable on the menu and optionally the shape of the menu itself In oneembodiment, the occlusion area 22 which does not contain any direct peninput areas extends for approximately 90 degrees, and more preferably 92degrees (out of 360 degrees of the circle). Other sized occlusion areasare possible as well. For example, in FIGS. 3-4 the occlusion areaextends almost 135 degrees. Since a human hand will occupy a relativelylarge angular space when placed close to the center of a circle,preferably the occlusion area is not less than about 75 degrees.

The functions to be placed on the menus may be selected such that morefrequently used functions are made available via the icons in regions 16and the less frequently used functions are available via sub-menus 26.For example, the “undo” function may be frequently used and thus may beincluded as a top level function (a function placed in one of theregions 16). Indeed, this function and possibly a small set of otherfunctions may be so frequently used by users that these functions may bepermanently located in particular regions 16 and not move, even if themenu supports scrolling as shown in FIGS. 9 and 10. Thus, the scrollingfunction shown in connection with FIGS. 9 and 10 need not cause allfunctions to scroll, but rather may have particular regions thatmaintain the same function even when the user interacts with the arrowor gesture region to cause the function of the other regions to change.

Depending on the resolution of the large digital surface, the size ofthe menu may need to be drawn relatively large to enable users to seethe icons with sufficient clarity to ascertain the function. When theusers are standing in front of a large digital surface such as anelectronic white-board, the use of large icons is less problematic sincethe user can simply use his arm to reach a necessary distance to selectthe correct functional icon. On a horizontal large digital surface, theuser is more likely to be resting his wrist on the surface of the largedigital surface. This makes it less convenient for the user to reach toaccess particular icons. To make the menu more convenient for use, thesize of the menu, including the diameter of the pie menu and thepercentage occlusion (amount of the area 22 that is not used for directpen input) may be customized for individual users to enable the users tospecify what their menu should look like to facilitate optimalinteraction between the user and the large digital surface.

According to an embodiment of the invention, a modified pie menu isprovided in which the selectable regions are ergonomically arranged toconform to the natural range of motion and reach of a user's hand. Sincethe size of users' hands and wrist flexibility differs greatly, many ofthe aspects of the ergonomically designed modified pie menu areconfigurable to more closely comport with the actual range of motion ofthe user's hand and wrist.

FIG. 13 shows the natural range of motion that is comfortable for thewrist of a right-handed person. A mirror image of FIG. 13 would show therange of motion of a person's left hand. As shown in FIG. 13, if aperson has their right wrist and forearm aligned generally along line130, and holds a pen in their right hand, an average person will be ableto use a pen to interact within the area defined by the two curved lines133, 134, in-between lines 136, and 138. Since people's hand sizes varygreatly and their wrist flexibility varies greatly as well, some peoplewill be able to reach close to the maximum top distance while otherswill only be able to reach to an area close to the minimum top distance.Likewise on the bottom, some people will be able to reach farther thanothers.

FIG. 14 shows another diagram of the natural range of motion of a personwith a stationary hand (reachable area from a fixed wrist position), aswould be expected of a user 16 seated at a large digital surface. Thisfigure shows the results of a study that was conducted by having peoplesit and draw on paper the range of motion that they could comfortablydraw without moving their hand.

Participants were seated at a table with large piece of paper in frontof them. The paper had a marked position for the wrist. All subjectswere asked to rest their hand on that spot, the direction of the arm wasnot predefined since the participants were intended to use a comfortablenatural position rather than a contrived position. The direction of theparticipant's arm was marked on the paper with a line and theparticipants were then asked to draw the maximum possible radius withoutstrain. Afterwards, they drew the minimum radius. The final taskinvolved sketching the reachable area with all accessible peaks. Arelatively small number of participants were used for the study, solarger groups of participants may result in slightly different valuesand more precise data.

In this study, the average hand direction angle was 148.28° (with astandard deviation (SD) of 11.05°) for right-handed and 28.17°(SD=13.28°) for left-handed users. The larger standard deviation forleft-handed users resulted from fewer left handed users in the study.The hand direction angle A1 is important to correctly place the pie menuon the screen.

The flexibility of the hand was calculated as the average of thereachable angles. An average angle B1, B2 of 82.19° (SD=26.63°) wascalculated for all users. In this case, left and right handed users werenot distinguished because the direction of the drawing is not relevantfor the angle.

The last parameters that define the area that can be reached are thepossible minimum and maximum distances, i.e. the distances defined bylines 132 and 134 in FIG. 13. For the outer arc 132, an average maximumbottom distance of 110.94 mm (SD=21.9 mm) was measured, and 131.12 mm(SD=30.0 mm) was measured for the average maximum top distance. Theinner arc 134 had an average minimum bottom distance of 69.05 mm(SD=16.41 mm) and an average minimum top distance of 79.74 mm (SD=23.89mm).

Based on these results, it was determined that there are menu designparameters that may be pre-configured and other menu design parametersthat should be customizable by the users. FIG. 15 shows these results.As shown in FIG. 15, the hand angle and, hence the occlusion area, maybe pre-configured for all users since the hand angle had a relativelysmall standard deviation. The other factors, such as the differencebetween the average minimum and average maximum reach (width) should becustomizable so that users with different sized hands can easily reachany menu created within the reachable area. Likewise since the wristflexibility limits may change for different users, the height of theusable area for drawing a menu should also be customizable.

The user may customize the height and width parameters when they firststart using a pen so that any menu drawn for the user is confined to fitwithin the defined parameters. There are many ways for a user to dothis. For example, the user may select between a plurality of menudrawing area sizes as shown in FIG. 16. In this example, the user may bepresented with a number of different menu size options such as a smallsize menu (left figure in FIG. 16), medium size menu (middle figure inFIG. 16), and large size menu (right figure in FIG. 16). By selectingthe size of the area in which any menu will be drawn, the user mayconfigure the menu so that all clickable icons drawn in a resultant menuwithin that area are within easy reach of the user without requiring theuser to move his hand.

FIG. 17 shows another way that the user may be allowed to define thesize of the menu that will be drawn. Specifically, in the example shownin FIG. 17 the user initially places the pen tip at an initial penposition. The user is then instructed to touch the pen tip to the tablein various positions shown using black dots. Depending on theimplementation fewer pen touches (fewer black dots) may be sufficient todetermine the active area in which to draw the menu. Likewise, ratherthan use pen touches, pen strokes may be used to determine the activearea in which to draw the menu instead. The places that the user touchesthe table may be used directly to define the menu area or may be used toselect one of a plurality of pre-configured menu areas for the user. Inthe example shown in FIG. 17, the menu area has been drawn to coincidewith the locations that the user touched the table. The system could,instead, use these pen touches to determine which of the areas shown inFIG. 16 should be used to draw a menu for the user.

FIG. 18 shows an open pie menu 180 according to an embodiment of theinvention that has been drawn within a reachable area 182 for a user.The open pie menu is formed to have a point-and-click regions 184 thatextend in an arc to follow the general shape of the outside of thereachable area. Icons 186 are drawn in the point-and-click regions 184to convey the function of the region to the user. A gesture area 188 isprovided as well and, in this design, is disposed within an area thatwould be expected to be occluded by the user's hand when the userinteracts with the menu. The gesture area may be used to cause thefunction of the point-and click regions within the interactive area tochange or to enable other types of gesture based interaction between theuser and the functions supported by the menu.

If the user selects one of the functions, and selection of the functionrequires a subfunction menu to be drawn for the user, the initial menuof functions may be replaced or, alternatively, the sub-function menumay be drawn over the top of the current function menu. Since the goalis to provide the user with an easy-to-reach palate of functions, thesub-function menu should also be drawn at least partially within thereachable area 182. Although a portion of the sub-function menu mayextend out of the reachable area, the sub-function menu should extend atleast partially into the reachable area so that each sub-functionincluded in the subfunction menu is able to be selected by the userwithout requiring the user to move his hand.

In the example shown in FIG. 18 the circle segment of functions includesa single row of point-and-click regions extending in an arc about theuser's hand. The invention is not limited in this manner as multiplerows of point-and-click regions may be included. For example, FIG. 19shows an embodiment with two rows of point-and-click regions. Thus, themenu may include one, two, or more rows of point-and-click regionssupporting functions available via the menu.

FIG. 20 shows an example in which a modified pie menu of FIG. 4 has beendrawn within the reachable region surrounding the initial pen placement.As shown in FIG. 20, the menu 10 and any sub-function menu 26 should bedrawn within the reachable area 182 to make those functions andsub-functions easily reachable for the user without requiring the userto move his hand.

As described in greater detail above, embodiments of the inventionutilize open pie-shaped menus that avoid placing interactive regions inareas that are expected to be occluded by the user's hand while the userinteracts with the menu. Depending on the handedness of the user,buttons that invoke an action that shows feedback in the pie menu shouldbe placed on the side of the menu that prevents occlusions of the menuitems. Additionally, according to an embodiment, the standardfunctionality of a menu including point-and-click regions is extended toinclude interactive gesture areas to enable gesture-based interactionbetween the user and the menu.

The gesture area interprets a set of gestures that further invokeactions. Since visual feedback is not required for the user to interactwith the gesture area, the gesture area can be placed in partly occludedregions. Preferably the gesture area is provided at the pen's initiallocation when the menu is invoked to facilitate instant use of gestures.

Submenus should not be drawn significantly outside the reachable area ofthe user and, accordingly, should be stacked or otherwise drawn to bereachable by the users. Likewise, the submenus should not be drawn inoccluded areas since interaction between the user and the submenurequires visual feedback for the user to select the proper sub-function.Further, since the position of occluded area depends on the location ofthe hand/pen, a placement below the parent item is favorable for bothvertical and horizontal displays to be visible to the user. Thus,according to one embodiment the sub-menus are drawn on top of theoriginal point-and-click region. Whenever the user is clicking to themain menu again, the submenu should fade out and disappear.

In addition to using menus formed directly on the large digital surface,a separate tangible palette may also be used to access a limited set ofmenu items. In this embodiment, instead of activating a pie menu onscreen and then selecting an option from the menu, the item is putdirectly on a shortcut palette. A tangible palette is a hardware devicecontaining an interactive screen via which the user may select functionsto be implemented on the large digital surface. The palette can be usedwith the same pen as the digital whiteboard, so that the user is notrequired to switch devices to select menu items on the separate tangiblepalette. The pie menu on the whiteboard still can be used with its fullfunctionality. The palette is an additional tool for frequentlyperformed actions that can be more quickly invoked without first havingto activate a pie menu.

FIG. 21 shows an example collaboration server 56 that may be used inconnection with an embodiment of the invention. In the embodiment shownin FIG. 21, the collaboration server is configured as a normal computerserver and includes a processor 200 and memory 202. The collaborationserver also includes other supporting circuitry such as networkinterface 201 and other common components (not shown).

In the embodiment shown in FIG. 21, the memory includes collaborationsoftware 206 and electronic pen software 208. These software componentscontain data and instructions which, when loaded into the processor 200,cause the processor 200 to implement collaboration process 204 andelectronic pen process 210. These processes allow the collaborationserver to receive input from electronic pen and incorporate the writingfrom the electronic pen into the collaboration session hosted by thecollaboration server via collaboration process 204. The collaborationprocess and electronic pen process further implement the functionsdescribed above in connection with establishing collaboration sessionsand enabling the users to interact with collaboration sessions usingelectronic pens.

FIG. 22 shows an example large digital surface 57 that may be used inconnection with an embodiment of the invention. In the embodiment shownin FIG. 22, the large digital surface is configured to include a largelight/touch sensitive digital surface 216 and an associated computer218. The large light/touch sensitive surface 216 provides a userinterface that can present information to the users as well as receiveinput from users. The surface 216 may be vertical and mounted on a wallto be used in place of a conventional white board, or may behorizontally mounted for example as a table surface.

The particular manner in which users can interact with the large lightand touch sensitive surface will depend on the technology used toimplement the digital surface . For example, if the surface 216 ispressure or light sensitive, users may use a stylus to press on thesurface or use a light pen 238 to interact with the surface. These typesof devices may be used to interact with content shown on the surface 216to select particular content, move content around, add content, andotherwise provide input via the surface 216 to the computer system.

The computer system 218 supporting the surface 216 may be implemented asa normal computer and includes a processor 220 and memory 222. Thecomputer 218 also includes other supporting circuitry such as networkinterface 224 and other components commonly utilized to enable thecomputer to operate appropriately (not shown).

In the embodiment shown in FIG. 22, the memory 222 includes applicationsoftware 228, collaboration software 232, and menu software 236. Thesesoftware components contain data and instructions which, when loadedinto the processor 220, cause the processor 220 to implement applicationprocess 226, collaboration process 230, and menu process 234. Theapplication process is a process that supports data collection andmanipulation on the surface 216. For example, application software 228may be a whiteboard software package to enable a whiteboard applicationprocess to be instantiated in processor 220. Users interact with thewhiteboard process by drawing on the surface 216, moving content, etc.The electronic whiteboard process in this instance provides theunderlying support to enable the surface to be used in this manner.Other application processes such as PhotoShop, PowerPoint, and othercommon applications may be utilized as well.

The collaboration 230 enables the large digital surface 57 to interactwith a collaboration server 56 to receive input from remote users. Theremote users, in this embodiment, are not physically located in the roomwith the surface 216 and hence cannot interact with the surface 216 bytouching the surface. Collaboration process 230 facilitatesnetwork-based interaction to enable remote users to interact with thecontent being shown on the surface as if they were physically present.For example, as discussed above, users may use digital pens to addcontent to the content being shown on the surface 216 and to movecontent around on the surface 216. Other types of remote interaction maybe supported as well, such as mouse and keyboard based interaction by auser at a remote terminal.

In the example shown in FIG. 22, the menu process 234 enables theprocessor to draw menus for users on the surface 216 upon occurrence ofa menu invoking action. The menu process has been shown as a separateprocess in this embodiment to separate the functionality associated withdrawing menus from the functionality associated with the underlyingapplication supporting the surface 216 or any applications running onthe surface 216. By separating the menu functionality into a separatecomponent the menus of those applications (which may be standard Windowsbar menus) are not required to be changed. Rather, the menu process canreplace the standard menus with the adaptive pie menus and adaptiveopen-pie menus described herein. Over time, the application processesmay incorporate the functionality of the menu process enabling thatcomponent or the functionality of that component to be implemented inother processes running on the processor 220.

The functions described above may be implemented as a set of programinstructions that are stored in a computer readable memory and executedon one or more processors on the computer platform. However, it will beapparent to a skilled artisan that all logic described herein can beembodied using discrete components, integrated circuitry such as anApplication Specific Integrated Circuit (ASIC), programmable logic usedin conjunction with a programmable logic device such as a FieldProgrammable Gate Array (FPGA) or microprocessor, a state machine, orany other device including any combination thereof. Programmable logiccan be fixed temporarily or permanently in a tangible medium such as aread-only memory chip, a computer memory, a disk, or other storagemedium. All such embodiments are intended to fall within the scope ofthe present invention.

It should be understood that various changes and modifications of theembodiments shown in the drawings and described in the specification maybe made within the spirit and scope of the present invention.Accordingly, it is intended that all matter contained in the abovedescription and shown in the accompanying drawings be interpreted in anillustrative and not in a limiting sense.

The invention claimed is:
 1. A collaboration system, comprising: anelectronic pen configured to enable a user to write on paper such thatwhat the user is writing while using the electronic pen is visible onthe paper, the electronic pen further being configured to recordinformation about strokes made by the user while writing; and acollaboration server including a storage medium, the collaborationserver configured to: establish a collaboration session displayable on adigital surface; receive input from the electronic pen over acommunication network and store the input from the electronic pen on thestorage medium; process the input from the electronic pen to recreatewhat the user was writing while using the electronic pen; and cause therecreated writing to be shown during the collaboration session on thedigital surface; wherein, in response to the user of the digital surfaceinvoking a menu, the digital surface is configured to display the menu,wherein the menu is a pie-shaped menu having an input area, the inputarea extending in a circular segment extending less than 360 degreessuch that no portion of the input area is disposed in an area expectedto be occluded by the user's hand when the user interacts with the menu.2. The collaboration system of claim 1, wherein the collaboration serveris further configured to store the recreated writing and thecollaboration session.
 3. The collaboration system of claim 1, whereinone or more applications are configured to be run on the digitalsurface, and wherein the menu is configured to control the one or moreapplications.
 4. The collaboration system of claim 3, further comprisinga hardware device comprising an interactive screen, and wherein users ofthe digital surface are able to invoke a menu associated with the one ormore applications by interacting with the interactive screen to causethe menu associated with the one or more applications to be drawn on thedigital surface.
 5. The collaboration system of claim 1, wherein themenu is invokable via the electronic pen with a single down action andtwo press actions within a defined period of time.
 6. The collaborationsystem of claim 1, wherein the menu is centered approximately at thelocation where the menu was invoked.
 7. The collaboration system ofclaim 1, wherein the area expected to be occluded extends approximately90 degrees.
 8. The collaboration system of claim 1, wherein anorientation of the area expected to be occluded is based on a directionvector between a location where the menu was invoked and a location ofcontact between the digital surface and the user's hand.
 9. Thecollaboration system of claim 8, wherein the location where the menu isinvoked is a tip of the electronic pen contacts the digital surface. 10.The collaboration system of claim 8, wherein, in response to the userinvoking the menu without any contact between the user's hand and thedigital surface, the orientation of the area expected to be occluded isestablished using a default value.
 11. The collaboration system of claim8, wherein, in response to the user invoking the menu without anycontact between the user's hand and the digital surface, the orientationof the area expected to be occluded is established using a previousdirection vector from a previous menu invocation.
 12. The collaborationsystem of claim 8, wherein the direction vector enables the areaexpected to be occluded to be adjusted based on the user's posture. 13.The collaboration system of claim 8, wherein the direction vectorenables the server to distinguish between right-handed users andleft-handed users, and enables the collaboration server to automaticallygenerate a right-handed menu for a right-handed user and a left-handedmenu for a left-handed user.
 14. The collaboration system of claim 1,wherein the menu further has a gesture area, configured to enable theuser to provide gesture based input.
 15. The collaboration system ofclaim 14, wherein the gesture area extends into the area expected to beoccluded.
 16. The collaboration system of claim 15, wherein the gesturearea enables functions associated with the input area to be movedbetween regions of the input area.
 17. The collaboration system of claim16, wherein the gesture area enables new functions to be associated withthe regions of the input area.
 18. The collaboration system of claim 1,wherein the user is remotely located from the digital surface.